Decision Theoretic Learning of Human Facial Displays

Changes in the human face occur due to many factors, including communication, emotion, speech, and physiology. Most systems for facial expression analysis attempt to recognize one or more of these factors, resulting in a machine whose inputs are video sequences or static images, and whose outputs are, for example, basic emotion categories. Our approach is fundamentally different. We make no prior commitment to some particular recognition task. Instead, we consider that the meaning of a facial display to an observer is contained in its relationship to actions and outcomes. Agents must distinguish facial displays according to their affordances, or how they help an agent to maximize utility. We show how an agent can learn relationships between observations of a person’s face, the context in which that person is acting, and its own actions and utility function. The agent can then base its decisions on these learned decision-theoretic models, allowing it to make value-directed action choices based, in part, upon observed facial displays. Recent research on the communicative function of the face supports our approach. Psychologists have concluded that facial displays are purposeful communicative signals [1], that the purpose is dependent on both the display and the context of its emission [4], and that the signals vary widely between individuals [4]. These considerations imply that a rational communicative agent must learn the relationships between facial displays, the context in which they are shown, and its own utility function: it must be able to compute the utility of taking actions in situations involving purposeful facial displays. The agent will then be able to make value-directed decisions based, in part, upon the “meaning” of facial displays as contained in these learned connections between displays, context, and utility. Learning these relationships will further allow an agent to adapt to new interactants and new situations. The model we propose is a partially observable Markov decision process, or POMDP, which realises the design constraints suggested by the psychology literature, combining the recognition of facial signals with their interpretation and use in a consistent utility-maximization framework. The parameters of the model are learned from training data using an a posteriori constrained optimization technique, such that an agent can learn to act based on the facial signals of a human through observation. The training is unsupervised: we do not train classifiers for individual facial displays, and then combine them in the model. Rather, the learning process discovers clusters of facial motions and their relationship to the context automatically. The advantage of this approach is threefold. First, we do not need expert knowledge about which facial motions are important. Second, since the system learns categories of motions, it will adapt to novel gestures or displays without modification. Third, resources can be focused on tasks that will be useful for the agent. It is wasteful to train complex classifiers for the recognition of fine facial motion if only simple displays are being used in the agent’s context.